Information processing apparatus, information processing method, and program

ABSTRACT

A plurality of feature values are extracted from input data as document data, distributed representations of words that correspond to the respective extracted plurality of feature values is obtained, and the extracted plurality of feature values are aggregated into a plurality of classifications based on the obtained distributed representation.

TECHNICAL FIELD

The present invention relates to an information processing apparatus, aninformation processing method, and a program.

BACKGROUND ART

Recently, an Artificial Intelligence (AI) has been actively studied anddeveloped, and has rapidly increased its practical applications. The AIis to artificially reproduce various perception and intelligence, suchas learning, inferencing, and determining, demonstrated by a human usinga computer.

Among AI, RPA (Robotic Process Automation, Digital Labor), which takesstates of working and decision-making by a knowledge worker as a modelto robotize them, is to achieve automation and efficiency of anoperation. In the RPA, the AI has evaluated a document.

Regarding such a technique to evaluate a quality of a document, there isproposed a technique to evaluate a quality of a document and present adocument that serves as a sample in Patent Literature 1.

Patent Literature 1 discloses a document quality evaluation system thatpresents a text to be improved and an example of a text to be a samplein addition to an evaluation result of a quality of a text documentgroup to an inputter. The document quality evaluation system performs asyntax analysis on each of texts in the text document group, scores aquality of the text document group on a plurality of evaluation items,presents the evaluation result to a first user, extracts a first textgroup that corresponds to a condition to cause a low evaluation in thelowest evaluation item for the first user and a second text group thatcorresponds to a condition to cause a high evaluation for a second userwho is evaluated higher than the first user in the lowest evaluationitem, extracts first and second texts high in similarity degrees fromthe first and second text groups, respectively, and presents the firsttext as an exemplary sentence subject to be improved and the second textas an exemplary sample sentence.

CITATION LIST Patent Literature

Patent Literature 1: Japanese Laid-open Patent Publication No.2011-170535

SUMMARY OF INVENTION Technical Problem

An evaluation of a document is performed based on feature valuesextracted from document data in some cases. While various feature valuescan be extracted from the document data, the feature values extractedfrom the document data have many similar feature values, and thus beingcomplicated. For example, even a user confirms these feature values, itis difficult to grasp what kind of document the document is. Therefore,there has been a demand to aggregate the similar feature valuesextracted from the document data.

Patent Literature 1 could not aggregate the similar feature valuesextracted from the document data.

Solution to Problem

Therefore, an information processing apparatus of the present inventionincludes an extractor, an obtainer, and an aggregator. The extractor isconfigured to extract a plurality of feature values from input data asdocument data. The obtainer is configured to obtain distributedrepresentations of words that correspond to the respective plurality offeature values extracted by the extractor. The aggregator is configuredto aggregate the plurality of feature values extracted by the extractorinto a plurality of classifications based on the distributedrepresentation obtained by the extractor.

Advantageous Effects of Invention

The present invention ensures aggregating similar feature valuesextracted from document data.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a drawing illustrating an exemplary hardware configuration ofan information processing apparatus.

FIG. 2 is a drawing illustrating an exemplary function configuration ofthe information processing apparatus.

FIG. 3 is a flowchart illustrating an exemplary evaluation criteriondetermination process.

FIG. 4 is a drawing illustrating an exemplary dendrogram indicatingclustering results.

FIG. 5A is a drawing illustrating an exemplary word presentation screen.

FIG. 5B is a drawing illustrating an exemplary word presentation screen.

FIG. 5C is a drawing illustrating an exemplary word presentation screen.

FIG. 5D is a drawing illustrating an exemplary word presentation screen.

FIG. 5E is a drawing illustrating an exemplary word presentation screen.

FIG. 5F is a drawing illustrating an exemplary word presentation screen.

FIG. 5G is a drawing illustrating an exemplary word presentation screen.

FIG. 5H is a drawing illustrating an exemplary word presentation screen.

FIG. 5I is a drawing illustrating an exemplary word presentation screen.

FIG. 5J is a drawing illustrating an exemplary word presentation screen.

FIG. 5K is a drawing illustrating an exemplary word presentation screen.

FIG. 5L is a drawing illustrating an exemplary word presentation screen.

FIG. 5M is a drawing illustrating an exemplary word presentation screen.

FIG. 5N is a drawing illustrating an exemplary word presentation screen.

FIG. 5O is a drawing illustrating an exemplary word presentation screen.

DESCRIPTION OF EMBODIMENTS

The following describes embodiments of the present invention based ondrawings.

Embodiment 1

(Outline of Process of this Embodiment)

An outline of a process of an embodiment will be described. In thisembodiment, an information processing apparatus 100 is assumed to be amain body of processing. The information processing apparatus 100 is aninformation processing apparatus, such as a personal computer (PC), aserver device, a tablet device, a smartphone, and the like.

In this embodiment, the information processing apparatus 100 extractsfeature values on which an individuality of a user, who is in charge ofevaluating ticket data, is reflected from ticket data (document data) inan object management system, and converts words corresponding to theextracted feature values into distributed representations. Then, theinformation processing apparatus 100 clusters the extracted featurevalues based on distances between words converted into the distributedrepresentations to aggregate the extracted feature values.

The information processing apparatus 100 accepts specifying clustersthat include the feature values used for determining the evaluationcriteria, and determines the evaluation criteria for the ticket data onwhich the individuality of the user, who is in charge of evaluating theticket data, is reflected based on the feature values included in theclusters indicated by the accepted specification.

(Hardware Configuration of Information Processing Apparatus)

FIG. 1 is a drawing illustrating an exemplary hardware configuration ofthe information processing apparatus 100.

The information processing apparatus 100 includes a CPU 101, a mainstorage unit 102, an auxiliary storage unit 103, a network I/F 104, andan input and output I/F 105. Each of the elements is communicativelycoupled to one another via a system bus 106.

The CPU 101 is a central processing unit that controls the informationprocessing apparatus 100. The main storage unit 102 is a storage device,such as a Random Access Memory (RAM) that functions as a work area forthe CPU 101 and a temporary storage location of data.

The auxiliary storage unit 103 is a storage device that stores varioussetting information, various programs, teacher data, various dictionarydata, various model information, and the like. The auxiliary storageunit 103 is configured as a storage medium for, for example, Read OnlyMemory (ROM), a hard disk drive (HDD), a solid state drive (SSD), aflash memory, and the like.

The network I/F 104 is an interface used for a communication with anexternal device via a network, such as internet and LAN. The input andoutput I/F 105 is an interface used for inputting information from aninput device, such as a computer mouse, a keyboard, and an operatingunit of a touch panel. The input and output I/F 105 is also an interfaceused for outputting information to an output device, such as a display,a display unit of a touch panel, a speaker, and the like.

The CPU 101 executes a process based on a program stored in theauxiliary storage unit 103, thereby achieving a function of theinformation processing apparatus 100 described later in FIG. 2, aprocess of a flowchart described later in FIG. 3, and the like.

(Function Configuration of Information Processing Apparatus)

FIG. 2 is a drawing illustrating an exemplary function configuration ofthe information processing apparatus 100.

The information processing apparatus 100 includes an analyzing unit 201,a learning unit 202, an extracting unit 203, an obtaining unit 204, anaggregating unit 205, an output unit 206, an accepting unit 207, adetermining unit 208, and an evaluation unit 209.

The analyzing unit 201 performs an analyzation, such as a morphologicalanalysis, dependency parsing, word categorizing, and determination of afeature value of a segment in document data (for example, labelingsemantic role on segment) on document data.

The learning unit 202 learns a classification model used for identifyinggood or bad in quality of ticket data based on teacher data of apositive example configured of ticket data preliminarily confirmed as agood quality by a user and teacher data of a negative example configuredof ticket data preliminarily confirmed as a bad quality by the user. Inthis embodiment, the teacher data is preliminarily stored in theauxiliary storage unit 103 and the like.

The learning unit 202 learns this classification model using, forexample, Naive Bayes, Random Forest, and the like. The teacher data isone example of input data.

The extracting unit 203 extracts feature values on which anindividuality of the user is reflected from feature values of theteacher data based on respective levels of contribution of the featurevalues of the teacher data in the classification model learned by thelearning unit 202.

The obtaining unit 204 converts words corresponding to the featurevalues extracted by the extracting unit 203 into distributedrepresentations.

The distributed representation is a technique to represent a word with areal vector of a plurality of dimensions (for example, a hundred tothree hundred dimensions). There exists, what is called, adistributional hypothesis that assumes that a meaning of a word in adocument is determined from peripheral words (context words). Assumingthe distributional hypothesis, a word can be represented as a vector inwhich each element indicates a probability of appearance of each contextword. Since words that serve as the context words are enormous (onetrillion or more), the size of this vector also becomes enormous (onetrillion dimensions or more). However, the most elements of this vectorare zero. Therefore, this vector can be compressed (for example,compressed to a size of a hundred dimensions). In the distributedrepresentation, assuming the distributional hypothesis, a word isrepresented as thus compressed vector.

The closer the meanings of the words are, the closer the vectors of thewords represented in the distributed representations become. With thisproperty, the vectors indicated by the distributed representations ofthe words corresponding to the respective feature values extracted bythe extracting unit 203 are close vectors as the meanings are close.

In this embodiment, the information processing apparatus 100 clustersthese feature values based on distances between the vectors indicated bythe distributed representations of the words corresponding to thefeature values extracted by the extracting unit 203 using this property.The distance between the vectors is an index to indicate a degree ofdifference between the vectors and is, for example, a distance betweenending points of two vectors when starting points of the vectors are atthe same point. With this, the information processing apparatus 100 cancluster the feature values extracted from the document data such thatones with similar meanings are included in an identical cluster. Thecluster is one example of a classification in which the feature valuesare aggregated.

The word represented in the distributed representation also has thefollowing property: that is, the property that the closer a vector thatindicates a difference between a word (1) and a word (2) and a vectorthat indicates a difference between a word (3) and a word (4) are, themore similar a relationship between the word (1) and the word (2) and arelationship between the word (3) and the word (4) are.

The aggregating unit 205 clusters the feature values extracted by theextracting unit 203 based on the words converted into the distributedrepresentations by the obtaining unit 204. That is, the aggregating unit205 clusters the feature values extracted by the extracting unit 203based on the distances between the vectors of the distributedrepresentations of the corresponding words, and determines a pluralityof clusters in which the feature values are aggregated.

The output unit 206 outputs information on each cluster determined bythe aggregating unit 205, information of the words corresponding to thefeature values included in each cluster, and the like.

The accepting unit 207 accepts specifying a cluster including thefeature values used for determining an evaluation criterion from theplurality of clusters determined by the aggregating unit 205.

The determining unit 208 determines the evaluation criterion for theticket data on which the individuality of the user is reflected based onthe feature values included in the cluster indicated by thespecification accepted by the accepting unit 207.

The evaluation unit 209 evaluates the ticket data using the evaluationcriterion determined by the determining unit 208.

(Evaluation Criterion Determination Process)

FIG. 3 is a flowchart that illustrates an exemplary evaluation criteriondetermination process.

At S301, the analyzing unit 201 performs an analysis process on teacherdata stored in the auxiliary storage unit 103. The analyzing unit 201performs an analysis process, such as a morphological analysis,dependency parsing, word categorizing, labeling a feature value (forexample, a semantic role) on each segment, on each ticket data in theteacher data. The morphological analysis is a process to decompose adocument into morphemes (smallest units having meaning in language) todetermine parts of speech and the like of the respective morphemes. Thedependency parsing is a process to determine which segment depends onwhich segment. The word categorizing is a process that refers to adictionary and the like that stores correspondence information betweenwords and categories to determine categories of words in a document. Thelabeling a semantic role on each segment is a process that analyzes astructure for a text in a document to label a role of a segment (forexample, an “agent,” an “object” and the like) in interpreting a meaningof the predicate on each segment depending on the predicate in the text.The analyzing unit 201, for example, uses a semantic role labeling toolto label a semantic role on a segment.

With the process at S301, the analyzing unit 201 extracts the featurevalues used for learning the classification model by the learning unit202. In this embodiment, it is assumed that the analyzing unit 201extracts semantic roles labeled on respective segments at S301 as thefeature values used for learning the classification model by thelearning unit 202. However, the analyzing unit 201 may extractcategories of words or words themselves as the feature values used forlearning the classification model by the learning unit 202. Theanalyzing unit 201 may extract combinations of the semantic roleslabeled on the respective segments, the categories of the words, thewords themselves, and the like as the feature values used for learningthe classification model by the learning unit 202.

At S302, the learning unit 202 learns the classification model thatidentifies good or bad in quality of the ticket data based on thefeature values extracted from the teacher data at S301.

The classification model learned at S302 is a model learned from theteacher data including positive example data preliminarily confirmed asthe positive example by a user and negative example data preliminarilyconfirmed as the negative example by the user. Therefore, it can beassumed that the larger a level of contribution in this classificationmodel is, the more the individuality of the user in evaluating theticket data is reflected on the feature values. The level ofcontribution is an index that indicates how much the feature valuecontributes in identifying good or bad in quality of the ticket data.

At S303, the extracting unit 203 extracts the preliminarily determinednumber N (for example, 30) pieces of feature values in order from thelargest levels of contribution in the classification model learned atS302 from the feature values extracted at S301, and thus, extracts thefeature values on which the individuality of the user in evaluating theticket data is reflected. In this embodiment, the extracting unit 203determines the level of contribution of a certain feature value based ona probability of appearance of a segment that has the feature value inthe positive example data and the negative example data in the teacherdata. For example, the extracting unit 203 determines a value obtainedby dividing the probability of appearance of the segment that has thefeature value in the positive example data in the teacher data by theprobability of appearance of the segment that has the feature value inthe negative example data as the level of contribution of the featurevalue. Then, the extracting unit 203 identifies N pieces of levels ofcontribution in order from the largest one among the obtained levels ofcontribution to extract the feature values corresponding to theidentified levels of contribution.

The extracting unit 203 may determine a variation amount of an errorrate when values (the number of appearance) of the feature valueextracted from the teacher data are shuffled between samples regarding acertain focused feature value as the level of contribution. Theextracting unit 203 may determine a Gini coefficient as the level ofcontribution.

The extracting unit 203 may extract ones having the level ofcontribution in the classification model learned at S302 equal to ormore than a preliminarily determined threshold value out of the featurevalues extracted at S301 as the feature values on which theindividuality of the user in evaluating the ticket data is reflected.

At S304, the obtaining unit 204 identifies a head word of the segmentcorresponding to each feature value extracted at S303. The head wordmeans a word that represents the segment, and is, for example, a nounpart in the segment, a root form of a verb part in the segment, and thelike. The obtaining unit 204, for example, uses the semantic rolelabeling tool to identify the head word from the segment. The head wordin the segment that corresponds to each feature value extracted at S303is one example of the word corresponding to each feature value extractedat S303.

At S305, the obtaining unit 204 obtains distributed representations ofthe words identified at S304. The obtaining unit 204, for example,obtains the distributed representations of the words identified at S304through an unsupervised learning (word2vec, fastText, and the like).

At S306, the aggregating unit 205 clusters the feature values extractedat S303 based on the distributed representations obtained at S305. Morespecifically, the aggregating unit 205 clusters the feature valuesextracted at S303 based on distances between vectors indicated by thedistributed representations obtained at S305. The vectors indicated bythe distributed representations have a property that the closer themeanings of the words are, the closer the vectors become. Therefore, theaggregating unit 205 can aggregate the similar feature values. Theaggregating unit 205, for example, generates a dendrogram describedlater in FIG. 4 as a result of the process at S306.

The aggregating unit 205, for example, uses a method for hierarchicalclustering, such as Nearest Neighbor (NN) method and Ward's method, tocluster the feature values extracted at S303. The aggregating unit 205may use a method for non-hierarchical clustering to cluster the featurevalues extracted at S303.

At S307, the output unit 206 outputs the result of the process at S306.The output unit 206, for example, outputs the result of the process atS306 by causing the display unit, such as a display coupled via theinput and output I/F 105 and a monitor of an external device coupled viathe network I/F 105 to display information indicative of the featurevalues determined at S307. The output unit 206 may, for example, outputthe result of the process at S306 in a format of a dendrogram asillustrated in FIG. 4.

FIG. 4 is a drawing illustrating an exemplary dendrogram (tree diagram)illustrating a progress of the clustering process at S306 in ahierarchical structure. Numerals written at ends of branches or branchpoints indicate cluster IDs. The clusters indicated by the IDs (15, 07,27, 09, . . . , 05, 03) at right ends of the dendrogram in FIG. 4 areclusters that correspond to the respective feature values extracted atS303. A cluster that is made by two joined clusters is a clustercorresponding to the feature value in which the feature values of thejoined two clusters are aggregated. For example, the cluster ID 34 is acluster made by joining the cluster with an ID of 15 and the clusterwith an ID of 07, and is a cluster corresponding to the feature value inwhich the feature value of the cluster with the ID of 15 and the featurevalue of the cluster with the ID of 07 are aggregated. The clustercorresponds to the feature value in which more similar clusters areaggregated when the number of joining is less. The cluster indicated byID (58) at a left end of the dendrogram is a cluster in which all theclusters at the right end are joined.

In this embodiment, the output unit 206 outputs the result of theprocess at S306 by displaying a specification screen that includes thedendrogram in FIG. 4 and is used for specifying a category including thefeature value used for determining the evaluation criterion on thedisplay unit.

When detecting, for example, an operation of placing a cursor over an IDon the specification screen, the output unit 206 displays a presentationscreen that presents the head word in the segment corresponding to thefeature value indicated by the cluster of the TD for which a selectionoperation is performed on the display unit. FIGS. 5A to O are drawingsillustrating exemplary presentation screens that present the head wordin the segment corresponding to the feature value indicated by thecluster of the ID. The presentation screens as in FIGS. 5A to O areexemplary information indicative of words corresponding to the featurevalues included in the cluster for each of the aggregated plurality ofclusters. In this embodiment, the information processing apparatus 100uses the teacher data that is document data in Japanese. Therefore, inthe presentation screens as in FIGS. 5A to O, Japanese words thatcorrespond to the feature values included in the corresponding clusteror words translated from those Japanese words into another language (forexample, English) are presented.

The output unit 206 increases the respective words in size on thepresentation screens in FIGS. 5A to O as the feature valuescorresponding to the words move close to a center of the cluster. Thecloser the feature value corresponding to the word is to the center ofthe cluster, the more average word to represent the cluster the wordbecomes. Therefore, the output unit 206 displays the word increased insize as the meaning of the word comes close to what the cluster means.With this, the user who visually perceives the presentation screen canunderstand what the cluster means more easily.

The output unit 206, for example, causes the presentation screen toinclude the preliminarily determined number of (for example, ten andtwenty) words in order from the closest corresponding feature value tothe center of the cluster. With this, the user who visually perceivesthe presentation screen can understand what the cluster means byvisually perceiving only the preliminarily determined number of wordswithout confirming all the words included in the cluster.

The user can confirm whether each cluster is a cluster in which theplurality of feature values are aggregated with unity or not, whilevisually perceiving the presentation screen corresponding to the clusterwith each ID.

The presentation screen in FIG. 5I is a presentation screen thatcorresponds to a cluster with an ID of 45 made by joining a cluster withthe ID of 05 corresponding to FIG. 5B, a cluster with the ID of 18corresponding to FIG. 5D. Viewing the presentation screen in FIG. 5I,each word is a word to indicate “a state change of a thing,” and therecan be seen a unity as a whole.

The presentation screen in FIG. 5J is a presentation screen thatcorresponds to a cluster with an ID of 47 made by joining a cluster withan ID of 03 corresponding to FIG. 5A and the cluster with the ID of 45corresponding to FIG. 5I. Viewing the presentation screen in FIG. 5J,each word is a word to indicate “a state change of a thing,” and therecan be seen a unity as a whole.

The presentation screen in FIG. 5O is a presentation screen thatcorresponds to a cluster with an ID of 56 made by joining the clusterwith the ID of 47 corresponding to FIG. 5J and a cluster with an ID of54 corresponding to FIG. 5N. Viewing the presentation screen in FIG. 5O,there can be seen that each word has no unity as a whole (for example,no concept that unites “mention” and “protrude” can be found).

The presentation screen in FIG. 5K is a presentation screen thatcorresponds to a cluster with an ID of 50 made by joining a cluster withan ID of 10 corresponding to FIG. 5C and a cluster with an ID of 41corresponding to FIG. 5H. Viewing the presentation screen in FIG. 5K,each word is a word that indicates a concept of “communication anddetermination,” and there can be seen a unity as a whole.

The presentation screen in FIG. 5G is a presentation screen thatcorresponds to a cluster with an ID of 35 made by joining a cluster withan ID of 21 corresponding to FIG. 5E and a cluster with an ID of 30corresponding to FIG. 5F. Viewing the presentation screen in FIG. 5G,each word is a word that indicates a concept of “evaluation,” and therecan be seen a unity as a whole.

The presentation screen in FIG. 5M is a presentation screen thatcorresponds to a cluster with an ID of 53 made by joining the clusterwith the ID of 35 corresponding to FIG. 5G and the cluster with the IDof 50 corresponding to FIG. 5K. Viewing the presentation screen in FIG.5M, each word is a word that indicates a concept of “recognition anddetermination,” and there can be seen a unity as a whole.

The presentation screen in FIG. 5N is a presentation screen thatcorresponds to a cluster with an ID of 54 made by joining a cluster withan ID of 51 corresponding to FIG. 5L and the cluster with the ID of 53corresponding to FIG. 5M. Viewing the presentation screen in FIG. 5N,there can be seen that each word has no unity as a whole (for example,no concept that unites “re-cover” and “appropriate” can be found).

The user can confirm the cluster in which the feature values areaggregated with unity, and grasp on what sort of viewpoint a quality ofthe ticket data is identified as good or bad.

The user performs a selection operation, such as clicking and tapping,on the ID of the cluster that includes the feature values used fordetermining the evaluation criterion of the ticket data. In thisembodiment, the user performs the selection operation on the ID of thecluster in which the feature values that are especially treated asimportant as indexes for good or bad in quality are aggregated. Whendetecting the selection operation on an ID of a cluster, the acceptingunit 207 accepts specifying the cluster with the ID on which theselection operation has been performed, and determines the featurevalues corresponding to the cluster corresponding to the ID on which theselection operation has been performed as the feature values used fordetermining the evaluation criterion of the ticket data.

At S308, the accepting unit 207 determines whether the specification ofthe cluster is accepted or not via the specification screen output atS307.

The accepting unit 207 advances the process to a process at S309 whendetermining that the specification of the cluster is accepted, andterminates the process in FIG. 3 when determining that the specificationof the cluster is not accepted.

At S309, the determining unit 208 determines the evaluation criterionfor the ticket data based on the feature values corresponding to thecluster indicated by the specification determined to have been acceptedat S308. For example, assume that the cluster indicated by thespecification determined to have been accepted at S308 is the clusterwith the ID of 47 in the dendrogram in FIG. 4. In this case, the featurevalues corresponding to the cluster indicated by the specificationdetermined to have been accepted at S308 are three, “with statechange—change in relationship—change in relationship (object),” “withstate change—positional change—change in positional relationship(physical),” and “with state change—positional change—positional change(physical).”

For example, the determining unit 208 determines the evaluationcriterion as follows. That is, the determining unit 208 identifiesappearance frequencies of these feature values in the positive exampledata in the teacher data, and presents to the user by outputting by, forexample, displaying the identified appearance frequencies on the displayunit. The user confirms the appearance frequencies of the respectivepresented feature values to examine what an evaluation aspect should be(for example, whether to determine with threshold value or not, and whatthe threshold value should be then). The user performs an operation viaan input device coupled via the input and output I/F 105 after theexamination to instruct the evaluation aspect to the informationprocessing apparatus 100.

The determining unit 208 determines the evaluation aspect based on theoperation via the input device coupled via the input and output I/F 105.The determining unit 208, for example, obtains the instruction thatindicates to perform the threshold determination and the value of thethreshold used then based on the operation via the input device coupledvia the input and output I/F 105. The determining unit 208 determinesthe evaluation criterion based on the obtained instruction, the obtainedthreshold value, and the feature values corresponding to the clusterindicated by the specification determined to have been accepted at S308.For example, assume that 1 is obtained as a threshold value to divide alow evaluation and a moderate evaluation, and 2 is obtained as athreshold value to divide the moderate evaluation and a high evaluation.In this case, the determining unit 208 determines an evaluationcriterion that evaluates an evaluation value as the low evaluation whenthe appearance number of the feature values (“with state change—changein relationship—change in relationship (object),” “with statechange—positional change—change in positional relationship (physical),”and “with state change—positional change—positional change (physical)”)is less than one time, evaluates an evaluation value as the moderateevaluation when the appearance number of the feature values is one timeor more and less than two times, and evaluates an evaluation value asthe high evaluation when the appearance number of the feature values istwo times or more.

The determining unit 208 may determine what sort of viewpoint theevaluation criterion is on for the determined evaluation criteria. Forexample, the user confirms the words in the presentation screencorresponding to the cluster, grasps what sort of concept each of theconfirmed words indicates, and input the information (for example,character string) indicative of the grasped concept into the informationprocessing apparatus 100 using the input device coupled via the inputand output I/F 105. The determining unit 208 gives a label of a naturelanguage onto the determined evaluation criterion such that a human caneasily understand “on what sort of viewpoint the evaluation criterionbased on the specified cluster is” based on the information input viathe input device coupled via the input and output I/F 105.

The evaluation aspect includes an aspect that determines the evaluationvalue based on, for example, whether a deviation value of the appearancenumber of each feature value is within a predetermined range or not,other than the threshold determination of the appearance number of eachfeature value.

The determining unit 208 may determine the evaluation criterion based onthe feature values corresponding to the cluster indicated by thespecification determined to have been accepted at S308 and apreliminarily determined evaluation aspect without accepting specifyingthe evaluation aspect from the user.

The number of the clusters of which specifications have been accepted atS308 may be one or may be plural. When the plurality of clusters arespecified, the determining unit 208 determines the evaluation criterionfor each cluster at S309.

(Evaluating Process)

The evaluation unit 209 evaluates newly input ticket data using theevaluation criterion determined in the process in FIG. 3.

Assume that the evaluation criterion determined by the evaluationcriterion determined at S309 is the evaluation criterion that evaluatesan evaluation value to be a low evaluation when the appearance number offeature values (for example, “with state change—change inrelationship—change in relationship (object),” “with statechange—positional change—change in positional relationship (physical),”and “with state change—positional change—positional change (physical)”)is less than one time, evaluates an evaluation value to be a moderateevaluation when the appearance number of feature values is one time ormore and less than two times, and evaluates an evaluation value to be ahigh evaluation when the appearance number of feature values is twotimes or more.

In this case, the evaluation unit 209 extracts the feature values fromthe input ticket data, and identifies how any feature values that relateto the evaluation criterion are included in each of the extractedfeature values. The evaluation unit 209 evaluates the ticket data bydetermining the evaluation value of the ticket data in accordance withthe evaluation criterion based on the identified number.

The evaluation unit 209 evaluates the ticket data for each evaluationcriterion when a plurality of the evaluation criteria are determined inthe process in FIG. 3.

Thus, the evaluation unit 209 evaluates the ticket data using theevaluation criterion determined in the process in FIG. 3, therebyensuring performing the evaluation on which the individuality of theuser playing a role of evaluating is reflected. With this, theinformation processing apparatus 100 can reduce a labor by the userplaying a role of evaluating to evaluate the ticket data.

The output unit 206, for example, outputs the evaluation result by theevaluation unit 209 by displaying it on the display unit. The outputunit 206 may output an example sentence when the result of theevaluation by the evaluation unit 209 has not been a preliminarilydetermined evaluation value (for example, high evaluation). This examplesentence information is preliminarily stored in the auxiliary storageunit 103. The output unit 206 may output advice information when theresult of the evaluation by the evaluation unit 209 has not been apreliminarily determined evaluation value. The advice information ispreliminarily stored in the auxiliary storage unit 103. With such aprocess, the output unit 206 can assist a creator of the ticket data ingenerating the ticket data. This example sentence information and theadvice information are one example of supporting information for thecreator of the ticket data.

The output unit 206, for example, outputs the evaluation result in thefollowing format. Excellent, Good, and Poor in the following example ofthe evaluation result indicate a high evaluation, a moderate evaluation,and a low evaluation, respectively. Excellent, Good, and Poor eachindicate evaluation values in different evaluation criteria. Thedescription after Good and Poor is one example of the advice informationfor the creator of the ticket data.

Evaluation Result Example

Good—Description relating to “location or time” may be insufficient.Increasing information on specific location and precise date and timemay be a good idea. Poor—It seems there are few descriptions relating to“determination and prediction.” Please increase description about howyou recognize and determine event. There may be a pattern whererecognition or determination is concluded as if it is a fact.Good—Description relating to “purpose” could not be detected. When theissued ticket relates to continuous improvement, how about writing apurpose, such as what you would like to achieve?Excellent—Preciselywritten about “object.”

(Effect)

As described above, in this embodiment, the information processingapparatus 100 obtained the distributed representations of the wordsincluded in the segment corresponding to the plurality of feature valuesextracted from the teacher data, and aggregated the extracted pluralityof feature values by clustering based on the obtained distributedrepresentations. The vectors indicated by the distributedrepresentations have the property to become close vectors as themeanings are close. With this, the information processing apparatus 100can aggregate the similar feature values extracted from the documentdata.

The information processing apparatus 100 learned the classificationmodel that identifies good or bad in quality of the ticket data based onthe teacher data including the positive example and the negativeexample. Then, the information processing apparatus 100 extracted theplurality of feature values on which the individuality of the user inevaluating the ticket data is reflected based on the levels ofcontribution of the feature values of the teacher data in the learnedclassification model. The information processing apparatus 100 clusteredand aggregated the extracted feature values to determine the pluralityof clusters. With this, the information processing apparatus 100 canaggregate the similar feature values for the feature values on which theindividuality of the user in evaluating the ticket data is reflected.

The information processing apparatus 100 accepted specifying the clusterincluding the feature values used for determining the evaluationcriterion of the ticket data among each of the aggregated clusters, anddetermined the evaluation criterion of the ticket data based on thefeature values included in the cluster indicated by the acceptedspecification. With this, the information processing apparatus 100 candetermine the evaluation criterion of the ticket data on which theindividuality of the user is reflected. While the individuality asinformation is implicit, and it is difficult to express, with theprocess of this embodiment, the information processing apparatus 100 candetermine the evaluation criterion of the ticket data on which theindividuality of the user is reflected. Further, the informationprocessing apparatus 100 evaluates the ticket data based on thedetermined evaluation criterion. With this, the information processingapparatus 100 can perform the evaluation on which the individuality ofthe user is reflected for the ticket data, thereby ensuring the reducedlabor by the user to directly evaluate the ticket data.

Modification

In this embodiment, the information processing apparatus 100 aggregatesthe feature values extracted from the teacher data, each of which is theticket data. However, the information processing apparatus 100 mayaggregate the feature values extracted from other data.

For example, the information processing apparatus 100 may extract thefeature values for each segment with the process similar to that at S301from a plurality of comments a certain user has posted in the past to aposting service of comments and the like. The information processingapparatus 100 identifies a head word in a segment corresponding to theextracted feature values, and obtains a distributed representation ofthe identified head word. The information processing apparatus 100clusters the extracted feature values based on the obtained distributedrepresentations to aggregate the feature values having a similarmeaning. The information processing apparatus 100 displays theinformation indicative of the aggregated result on the display unit as adendrogram as in FIG. 4 that includes the presentation screens in FIG.5. With this, the user can grasp the individuality expressed in thecomments posted by him or herself. For example, the user can grasp whatsort of opinion he or she often posts to other people and the like.

In this embodiment, the evaluation unit 209 of the informationprocessing apparatus 100 evaluates the newly input ticket data using theevaluation criterion determined in the process in FIG. 3.

However, the AI implemented in the information processing apparatus 100may evaluate the newly input ticket data using the evaluation criteriondetermined in the process in FIG. 3. The AI implemented in an externalinformation processing apparatus may evaluate the newly input ticketdata using the evaluation criterion determined in the process in FIG. 3by the information processing apparatus 100.

Embodiment 2

While in a conversation, such as a conventional meeting and discussion,a width of topic is limited within a range of ideas of the participants,when the range of ideas is small, it is possible that the argument getsstuck due to the lack of ideas or reworking occurs due to an oversightof risks.

Therefore, in order to yield ideas beyond a frame of a team of theparticipants, there has been the demand that desires to make a knowledgeother than the knowledge of the current team, such as the knowledge of ateam in the past, easily available.

Therefore, in this embodiment, a description will be given of a processin which the information processing apparatus 100 presents a viewpointon which the conversation is possibly insufficient to a team that holdsa meeting.

The hardware configuration and the function configuration of theinformation processing apparatus 100 in this embodiment is similar tothose of Embodiment 1.

In this embodiment, the auxiliary storage unit 103 is assumed topreliminarily store minutes data (for example, document data, such astext data and voice data) that indicates contents of meetings held inthe past. The minutes data is assumed to include data of positiveexample (positive example data) preliminarily confirmed to be anappropriate content by a user and data of negative example (negativeexample data) preliminarily confirmed to be an inappropriate content bythe user.

An evaluation criterion determination process of this embodiment will bedescribed with reference to FIG. 3. The process at S301 to S307 of thisembodiment is similar to that of Embodiment 1 except that the minutesdata is used instead of the ticket data.

In this embodiment, the information processing apparatus 100 is assumedto determine a plurality of evaluation criteria. The informationprocessing apparatus 100, for example, determines the plurality ofevaluation criteria by repeating the process at S308 to S309 for severaltimes. The information processing apparatus 100, for example, may acceptspecifying a plurality of clusters at S308, and may determine theevaluation criterion based on the corresponding feature value for eachof the plurality of clusters at S309. The information processingapparatus 100, for example, may determine the evaluation criterion basedon the corresponding feature value for each of all the clusters at S309,assuming that specifying all the clusters is accepted at S308.

Assume that a certain team holds a meeting, and the minutes data of themeeting is newly input into the information processing apparatus 100 andis specified as an evaluation object.

The evaluation unit 209 evaluates the newly input minutes data usingeach of the plurality of evaluation criteria determined in the processin FIG. 3. The evaluation unit 209 determines a plurality of evaluationresults corresponding to the respective plurality of evaluation criteriadetermined in the process in FIG. 3.

In this embodiment, each of the plurality of evaluation resultscorresponding to the respective plurality of evaluation criteriadetermined in the process in FIG. 3 is represented in a value thatindicates the larger the better and the smaller the poorer.

The evaluation unit 209, for example, selects the preliminarilydetermined number of evaluation results in order from the smallest valueamong the plurality of these evaluation results. The evaluation unit209, for example, may select the preliminarily determined number ofevaluation results in order from the smallest value among the evaluationresults equal to or lower than a preliminarily determined thresholdvalue included in the plurality of these evaluation results. Theevaluation unit 209 identifies the evaluation criteria that correspondto the respective selected evaluation results, and identifies clustersused for determining the identified evaluation criteria.

The output unit 206, for example, sorts the clusters identified by theevaluation unit 209 in order from the lowest corresponding evaluationresult, and outputs the clusters by displaying them on the display unit.

As described above, in this embodiment, the information processingapparatus 100 determined the plurality of evaluation criteria based onthe past minutes data, and the document data as the evaluation objectwas evaluated using the determined plurality of evaluation criteria. Theinformation processing apparatus 100 selected a part of the evaluationresults in order from the lowest one, and output the clusterscorresponding to the evaluation criteria corresponding to the selectedevaluation results. With this, the information processing apparatus 100outputs the clusters that indicate viewpoints not appropriately includedin the document data as the evaluation object, thereby ensuringpresenting the viewpoints to a user belonging to the team holding themeeting. The user confirms the insufficient conversation on theviewpoints indicated by the presented clusters, thereby ensuring furtherappropriately conducting the subsequent conversation. Thus, theinformation processing apparatus 100 can make the knowledge other thanthe knowledge of the current team easily available.

The information processing apparatus 100 can continue keeping theknowledge of a person, for example, even though the person who held ameeting in the past has resigned, in order to make the knowledge otherthan the knowledge of the current team easily available without a human.The more the meeting is held, the more the available minutes dataincreases, and thus, the information processing apparatus 100 can makefurther more knowledge available.

Embodiment 3

In this embodiment, a description will be given of a process in whichthe information processing apparatus 100 presents a viewpoint on whichthe conversation is possibly insufficient to a team that holds a meetingwith a method different from that of Embodiment 2.

The hardware configuration and the function configuration of theinformation processing apparatus 100 in this embodiment is similar tothose of Embodiment 1.

In this embodiment, the auxiliary storage unit 103 is assumed topreliminarily store minutes data that indicates contents of the meetingsheld in the past similarly to Embodiment 2. In this embodiment, theminutes data preliminarily stored in the auxiliary storage unit 103 isdata having no distinction between positive example data preliminarilyconfirmed to be an appropriate content by a user and negative exampledata preliminarily confirmed to be an inappropriate content by the user.

The process of this embodiment will be described.

In this embodiment, the obtaining unit 204 extracts words from teacherdata using the past minutes data preliminarily stored in the auxiliarystorage unit 103 as the teacher data, and obtains distributedrepresentations of the extracted words. The aggregating unit 205clusters the words extracted by the obtaining unit 204 based on thedistributed representations obtained by the obtaining unit 204. Each ofthe obtained clusters as a result of this clustering is a word clusterthat serves as feature values of the words. The word clusters are thefeature values that indicate the respective plurality of clustersobtained by clustering the plurality of words. The word cluster as thefeature value of a certain word indicates the cluster to which the wordbelongs. The aggregating unit 205 determines arranged words as a name ofeach word cluster for each of the word clusters. The arranged words havea preliminarily determined number of words that are the closest to thecenter of the cluster.

The output unit 206 outputs results of clustering by the aggregatingunit 205. The output unit 206 may, for example, output the results ofclustering by the aggregating unit 205 in a format of a dendrogram asillustrated in FIG. 4.

Afterwards, assume that a certain team holds a meeting, the minutes dataof the meeting is newly input into the information processing apparatus100, and is specified as an evaluation object.

The analyzing unit 201 extracts all the words included in the specifiedminutes data, and extracts the feature values (word clusters) for therespective extracted words. The analyzing unit 201, for example, obtainsdistributed representations of the respective extracted words, andextracts the word clusters of the respective words based on whichclusters the obtained distributed representations belong to.

The analyzing unit 201 obtains an index that indicates how often thewords corresponding to the feature values appear in the minutes data asthe evaluation object for each feature value (word cluster). In thisembodiment, a discussion rate defined as follows is used as this index.The discussion rate is an index defined as (a total number of wordscorresponding to a certain feature value included in minutes data as anevaluation object)/(a total number of words corresponding to the featurevalue included in teacher data (minutes data preliminarily stored in theauxiliary storage unit 103).

The analyzing unit 201 obtains the discussion rate for each featurevalue (word cluster). It can be interpreted that the higher thediscussion rate is, the better the viewpoint corresponding to thefeature value is discussed on. It can be interpreted that the lower thediscussion rate is, the less sufficient the viewpoint corresponding tothe feature value is discussed on.

The analyzing unit 201 selects the preliminarily determined number ofdiscussion rates in order from the smallest value among the obtaineddiscussion rates. The analyzing unit 201 may, for example, select thepreliminarily determined number of discussion rates in order from thesmallest value among the discussion rates equal to or less than apreliminarily determined threshold value included in the obtaineddiscussion rates. The analyzing unit 201 identifies the feature values(word clusters) corresponding to the selected discussion rates.

The output unit 206 sorts the feature values identified by the analyzingunit 201 into order from the lowest corresponding discussion rate andoutputs them by displaying them on the display unit. With this, theoutput unit 206 can present the viewpoints on which the argument isinterpreted as insufficient to the user. The output unit 206 may outputa screen that presents the words (words included in word clusters)corresponding to the feature values identified by the analyzing unit 201by displaying them on the display unit. With this, the output unit 206can present the viewpoints on which the argument is interpreted asinsufficient to the user in more details.

In this embodiment, the discussion rate as the index to indicate howoften the words corresponding to the feature values appear in theminutes data as the evaluation object for each feature value (wordcluster) was defined as (a total number of words corresponding to acertain feature value included in minutes data as an evaluationobject)/(a total number of words corresponding to the feature valueincluded in teacher data (minutes data preliminarily stored in theauxiliary storage unit 103). However, the discussion rate may be anindex defined as described below.

First, for a word group (a collection of a plurality of words), a volumeof the word group is defined as follows. That is, a volume (hypervolume)of a convex hull in a vector space (semantic space) of distributedrepresentations is defined as the volume of the word group. Using pointsselected from a plurality of points indicating vectors corresponding tothe respective words included in a certain word group as respectiveapexes, the convex hull contains all the points other than the pointsused as the apexes among the plurality of points.

The discussion rate may, for example, be an index defined as follows byusing this volume of word group. That is, the discussion rate may be anindex defined as (a volume of a word group as a collection of all wordscorresponding to a certain feature value included in minutes data as anevaluation object)/(a volume of a word group as a collection of allwords corresponding to the feature value included in teacher data(minutes data preliminarily stored in the auxiliary storage unit 103).

In such a case, the analyzing unit 201 obtains the discussion rate asfollows. That is, the analyzing unit 201, first, identifies a word groupas a collection of all words that correspond to a certain feature valueincluded in minutes data as an evaluation object, and identifies pointson semantic space of all the words included in the identified wordgroup. The analyzing unit 201, using points included in the identifiedpoints as apexes, identifies a convex hull on the semantic spacecontaining all except for the points used as the apexes among theidentified points, and obtains a volume (hypervolume) of the identifiedconvex hull as the volume of the word group.

Next, the analyzing unit 201 identifies a word group as a collection ofall words corresponding to the feature value included in minutes datapreliminarily stored in the auxiliary storage unit 103, and identifiespoints on semantic space of all the words included in the identifiedword group. The analyzing unit 201, using the points included in theidentified points as apexes, identifies a convex hull on the semanticspace that contains all except for the points used as the apexes amongthe identified points, and obtains a volume (hypervolume) of theidentified convex hull as the volume of the word.

The analyzing unit 201 obtains the discussion rate by, for example,dividing the volume of the words obtained from the minutes data as theevaluation object by the volume of the words obtained from the minutesdata, which is preliminarily stored in the auxiliary storage unit 103,as the teacher data.

In this embodiment, the minutes data preliminarily stored in theauxiliary storage unit 103 was the data having no distinction betweenthe positive example data preliminarily confirmed to be the appropriatecontent by the user and the negative example data preliminarilyconfirmed to be the inappropriate content by the user. However, theminutes data preliminarily stored in the auxiliary storage unit 103 maybe data having a distinction between the positive example datapreliminarily confirmed to be the appropriate content by the user andthe negative example data preliminarily confirmed to be theinappropriate content by the user.

In such a case, the information processing apparatus 100, for example,performs the following process.

The information processing apparatus 100 performs a process similar tothat of Embodiment 2 up to the process at S307, except that the wordcluster is used as the feature value extracted at S301 and used in thesubsequent process.

Assume that, afterwards, a certain team holds a meeting, and the minutesdata of the meeting is newly input into the information processingapparatus 100, and is specified as the evaluation object.

The analyzing unit 201 extracts all words included in the specifiedminutes data, and extracts feature values (word clusters) for each ofthe extracted words. The analyzing unit 201 obtains a discussion ratethat indicates how often the words corresponding to the feature valuesappear in the minutes data as the evaluation object for each featurevalue (word cluster). The discussion rate in this case is, for example,obtained as (a total number (volume) of words corresponding to a certainfeature value included in minutes data as an evaluation object)/(a totalnumber (volume) of words corresponding to the feature value included inteacher data (past minutes data subject for the feature value extractionat S301)).

The analyzing unit 201, for example, obtains the discussion rate foreach feature value (word cluster), and selects the preliminarilydetermined number of discussion rates in order from the smallest valueamong the obtained discussion rates. The analyzing unit 201 may, forexample, select the preliminarily determined number of discussion ratesin order from the smallest value among the discussion rates equal to orless than a preliminarily determined threshold value included in theobtained discussion rates. The analyzing unit 201 identifies the featurevalues (word clusters) corresponding to the selected discussion rates.The output unit 206 sorts the feature values identified by the analyzingunit 201 into order from the lowest corresponding discussion rate, andoutput them by displaying them on the display unit. The output unit 206may output the word groups corresponding to the respective featurevalues.

When the minutes data preliminarily stored in the anexiliary storageunit 103 has a distinction between the positive example data and thenegative example data, the information processing apparatus 100 mayperform the process described above.

The minutes data preliminarily stored in the auxiliary storage unit 103has a distinction between the positive example data and the negativeexample data, the information processing apparatus 100 may perform thefollowing process.

The obtaining unit 204 identifies a collection of words excluding wordsincluded in the negative example data from a collection of wordsincluded in the positive example data. The obtaining unit 204 extractsthe words from the collection of the identified words, and obtainsdistributed representations of the extracted words. The aggregating unit205 clusters the words included in the collection of the identifiedwords based on the distributed representations obtained by the obtainingunit 204. As a result of this clustering, each of the obtained clustersis set to be a word cluster as the feature value. The aggregating unit205 generates a dendrogram as in FIG. 4 as a result of the process ofthe clustering.

Then, the output unit 206 outputs the result of the clustering by theaggregating unit 205.

Assume that, afterwards, a certain team holds a meeting, and the minutesdata of the meeting is newly input into the information processingapparatus 100, and is specified as the evaluation object.

The analyzing unit 201 extracts all words included in the specifiedminutes data, and extracts feature values (word clusters) for each ofthe extracted words. The analyzing unit 201 obtains a discussion ratethat indicates how often the words corresponding to the feature valueappear in the minutes data as the evaluation object for each featurevalue (word cluster). The discussion rate in this case is, for example,obtained as (a total number (volume) of words corresponding to a certainfeature value included in minutes data as an evaluation object)/(a totalnumber (volume) of words corresponding to the feature values included inteacher data (positive example data included in past minutes datapreliminarily stored in the auxiliary storage unit 103).

The analyzing unit 201, for example, obtains the discussion rate foreach feature value (word cluster), and selects the preliminarilydetermined number of discussion rates in order from the smallest valueamong the obtained discussion rates. The analyzing unit 201 may, forexample, select the preliminarily determined number of discussion ratesin order from the smallest value among the discussion rates equal to orless than a preliminarily determined threshold value included in theobtained discussion rates. The analyzing unit 201 identifies the featurevalues (word clusters) corresponding to the selected discussion rates.The output unit 206 sorts the feature values identified by the analyzingunit 201 in order from the lowest corresponding discussion rate, andoutputs them by displaying them on the display unit. The output unit 206may also output the word groups corresponding to the respective featurevalues.

When the minutes data preliminarily stored in the auxiliary storage unit103 includes the positive example data and the negative example data,the information processing apparatus 100 may perform the processdescribed above.

When the minutes data preliminarily stored in the auxiliary storage unit103 has a distinction between the positive example data and the negativeexample data, the information processing apparatus 100 may perform thefollowing process.

The obtaining unit 204 identifies a collection of words excluding wordsincluded in the positive example data from a collection of wordsincluded in the negative example data. The obtaining unit 204 extractswords from a collection of the identified words, and obtains distributedrepresentations of the extracted words. The aggregating unit 205clusters the words included in the collection of the identified wordsbased on the distributed representations obtained by the obtaining unit204. As a result of this clustering, each of the obtained clusters isset to be a word clusters as the feature value. In this case, the wordclusters as the feature values are clusters of words that appear only inthe negative example data and do not appear in the positive exampledata. Therefore, it can be interpreted that the higher the frequency ofappearance of the words having these feature values, the more theconversation on inappropriate viewpoints is conducted.

The aggregating unit 205 generates a dendrogram as in FIG. 4 as a resultof the process of the clustering. The output unit 206 outputs the resultof the clustering by the aggregating unit 205.

Assume that, afterwards, a certain team holds a meeting, and the minutesdata of the meeting is newly input into the information processingapparatus 100, and is specified ad the evaluation object.

The analyzing unit 201 extracts all words included in the specifiedminutes data, and extracts feature values (word clusters) for each ofthe extracted words. The analyzing unit 201 obtains a discussion rateindicating how often the words corresponding to the feature valuesappear in the minutes data as the evaluation object for each featurevalue (word cluster). The discussion rate in this case is, for example,obtained as (a total number (volume) of words corresponding to a certainfeature value included in minutes data as an evaluation object)/(a totalnumber (volume) of words corresponding to the feature value included inteacher data (negative example data included in past minutes datapreliminarily stored in the auxiliary storage unit 103).

The analyzing unit 201, for example, obtains the discussion rate foreach feature value (word cluster), and selects the preliminarilydetermined number of discussion rates in order from the highest valueamong the obtained discussion rates. The analyzing unit 201 may, forexample, select the preliminarily determined number of discussion ratesin order from the highest value among the discussion rates equal to ormore than a preliminarily determined threshold value included in theobtained discussion rates. The analyzing unit 201 identifies the featurevalues (word clusters) corresponding to the selected the discussionrates. The output unit 206 sorts the feature values identified by theanalyzing unit 201 into order from the highest corresponding discussionrate, and outputs them by displaying them on the display unit. Theoutput unit 206 may also output the word groups corresponding to therespective feature values.

With this, the information processing apparatus 100 can presentinappropriate viewpoints that are discussed to the user, and call user'sattention on the fact that the conversation is continued on suchviewpoints.

When the minutes data preliminarily stored in the auxiliary storage unit103 includes the positive example data and the negative example data,the information processing apparatus 100 may perform the processdescribed above.

As described above, in this embodiment, the information processingapparatus 100 obtained the discussion rate for each of the extractedfeature values, and identified the feature values indicating theviewpoints (such as viewpoint on which argument is interpreted asinsufficient and viewpoint on which inappropriate argument has beenmade) on which an attention should be paid in the conversationcorresponding to the minutes data as the evaluation object based on theobtained discussion rates. The information processing apparatus 100presented the identified feature values to the user who belongs to theteam holding the meeting by outputting them. With this, the user cangrasp the viewpoint on which the attention should be paid. The user canfurther appropriately conduct the subsequent conversation after graspingthe viewpoint indicated by the presented feature values. Thus, theinformation processing apparatus 100 can make the knowledge other thanthe knowledge of the current team easily available.

OTHER EMBODIMENTS

In Embodiments 1 to 3, the information processing apparatus 100 was aninformation processing apparatus alone. However, the informationprocessing apparatus 100 may be configured as a system including aplurality of the information processing apparatuses communicativelycoupled to one another via a network (LAN and Internet). In such a case,the respective CPUs of the plurality of information processingapparatuses included in the information processing apparatus 100 executethe processes in cooperation based on programs stored in auxiliarystorage units of the respective information processing apparatuses, andthus, the function in FIG. 2, the process in the flowchart in FIG. 3,and the like are achieved.

In Embodiments 1 and 2, the information processing apparatus 100clustered the plurality of feature values based on the distributedrepresentations of the words corresponding to each of the featurevalues, determined the evaluation criteria based on the clusteringresults, and output the feature values indicating the viewpoints onwhich the conversation is insufficient and the words corresponding tothe feature values as the evaluation results based on the determinedevaluation criteria, thereby presenting it to the user.

However, the information processing apparatus 100 may use a segment anda sentence, a paragraph (string of sentence), a document (string ofparagraph (string of sentence))(hereinafter, referred to as a segmentand the like) instead of a word, and use a vector obtained by using, forexample, the neural language model (existing method of deep learning)and the like from the segment and the like instead of the distributedrepresentation of the word. Using the neural language model and thelike, the vector that corresponds to the segment and the like having aproperty similar to that of the distributed representation of the wordthat becomes a close vector to one another as the meanings of each otherare close for the segment and the like can be obtained. Hereinafter,such a vector is a segment and the like vector.

In such a case, the information processing apparatus 100 clusters theplurality of feature values based on the segment and the like vector ofthe segment and the like corresponding to each of the feature values,determines the evaluation criteria based on the clustering results, andoutputs the feature values indicating the aspects on which theconversation is insufficient and the segment and the like correspondingto the feature values as the evaluation results based on the determinedevaluation criteria, thereby presenting it to the user.

In Embodiments 2 and 3, the description has been made of the process inwhich the information processing apparatus 100 determines the wordclusters as the feature values by clustering the words included in thepast minutes data, obtains the discussion rate for each feature value(word cluster), and outputs the feature values corresponding to thediscussion rates selected from the large (small) ones among the obtaineddiscussion rates and the words corresponding to the feature values.However, the information processing apparatus 100 may use the segmentand the like instead of the word and use the segment and the like vectorinstead of the distributed representation of the word.

In such a case, the information processing apparatus 100 may determinesegment and the like clusters as the feature values by clustering thesegment and the like included in the past minutes data, obtains thediscussion rate for each of the feature values (the segment and the likecluster), and output the feature values corresponding to the discussionrates selected from the large (small) ones among the obtained discussionrates and the segment and the like corresponding to the feature values.In this case, the discussion rate is, for example, obtained as (a totalnumber (volume) of segment and the like corresponding to a certainfeature value included in minutes data as an evaluation object)/(a totalnumber (volume) of segment and the like corresponding to the featurevalue included in teacher data (past minutes data preliminarily storedin the auxiliary storage unit 103)).

While preferable embodiments of the present invention have beendescribed above in details, the present invention is not limited to suchspecific embodiments.

For example, a part or all of the function configuration of theabove-described information processing apparatus 100 may be implementedin the information processing apparatus 100 as hardware.

1. An information processing apparatus comprising: an extractorconfigured to extract a plurality of feature values from input data asdocument data; an obtainer configured to obtain distributedrepresentations of words that correspond to the respective plurality offeature values extracted by the extractor; an aggregator configured toaggregate the plurality of feature values extracted by the extractorinto a plurality of classifications based on the distributedrepresentation obtained by the extractor; an acceptor configured toaccept a specification of a classification including a feature valueused for determining an evaluation criterion for document data among theplurality of classifications aggregated by the aggregator; and adeterminer configured to determine the evaluation criterion based on afeature value included in a classification Indicated by thespecification accepted by the acceptor.
 2. The information processingapparatus according to claim 1, further comprising a first outputterconfigured to output information indicative of the plurality ofclassifications aggregated by the aggregator.
 3. (canceled)
 4. Theinformation processing apparatus according to claim 1, furthercomprising an evaluator configured to evaluate document data based onthe evaluation criterion determined by the determiner.
 5. Theinformation processing apparatus according to claim 4, furthercomprising a second outputter configured to output a result of theevaluation by the evaluator.
 6. The information processing apparatusaccording to claim 5, wherein the second outputter further outputssupporting information for a creator of document data when the result ofthe evaluation by the evaluator is a preliminarily determined evaluationvalue.
 7. The information processing apparatus according to claim 4,wherein the evaluator evaluates document data based on each of aplurality of the evaluation criteria determined by the determiner, andthe information processing apparatus further comprises a third outputterconfigured to output a classification included in the plurality ofclassifications selected based on a plurality of evaluation results bythe evaluator that correspond to a respective plurality of theevaluation criteria determined by the determiner.
 8. The informationprocessing apparatus according to claim 3, wherein the acceptor furtheraccepts a specification of an evaluation aspect in the evaluationcriterion, and the determiner determines the evaluation criterion basedon the classification and the evaluation aspect indicated by thespecification accepted by the acceptor.
 9. The information processingapparatus according to claim 3, wherein the acceptor accepts aspecification of a classification that includes a feature value used fordetermining the evaluation criterion among the plurality ofclassifications aggregated by the aggregator via a specification screenused for specifying a classification including a feature value used fordetermining the evaluation criterion.
 10. The information processingapparatus according to claim 9, wherein the specification screenincludes information indicative of a word that corresponds to a featurevalue included in a classification for each of the plurality ofclassifications aggregated by the aggregator.
 11. The informationprocessing apparatus according to claim 1, wherein the input dataincludes positive example data and negative example data, the positiveexample data being confirmed to be a positive example by a user, thenegative example data being confirmed to be a negative example by theuser, and the extractor extracts the plurality of feature values basedon levels of contribution of respective feature values of the input datain a classification model learned based on the input data and used foridentifying whether document data is a positive example or a negativeexample.
 12. The information processing apparatus according to claim 11,wherein the extractor extracts a preliminarily determined number offeature values in order from a largest level of contribution as theplurality of feature values based on levels of contribution ofrespective feature values of the input data in the classification model.13. The information processing apparatus according to claim 11, whereinthe extractor extracts feature values of which levels of contributionare equal to or more than a preliminarily determined threshold value asthe plurality of feature values based on levels of contribution ofrespective feature values of the input data in the classification model.14. The information processing apparatus according to claim 1, furthercomprising a fourth outputter configured to output a feature valueselected based on an index indicating how often each of wordscorresponding to the respective plurality of feature values extracted bythe extractor appears in specified document data.
 15. The informationprocessing apparatus according to claim 1, further comprising a fifthoutputter configured to output a word that corresponds to a featurevalue selected based on an index indicating how often each of wordscorresponding to the respective plurality of feature values extracted bythe extractor appears in specified document data.
 16. An informationprocessing method executed by an information processing apparatus, themethod comprising: an extracting step of extracting a plurality offeature values from input data as document data; an obtaining step ofobtaining distributed representations of words that correspond to therespective plurality of feature values extracted at the extracting step;an aggregating step of aggregating the plurality of feature valuesextracted at the extracting step into a plurality of classificationsbased on the distributed representation obtained at the extracting step;an accepting step of accepting a specification of a classificationincluding a feature value used for determining an evaluation criterionfor document data among the plurality of classifications aggregated atthe aggregating step; and a determining step of determining theevaluation criterion based on a feature value included in aclassification indicated by the specification accepted at the acceptingstep.
 17. (canceled)
 18. A computer-readable recording medium thatrecords a program for causing a computer to execute: an extracting stepof extracting a plurality of feature values from input data as documentdata; an obtaining step of obtaining distributed representations ofwords that correspond to the respective plurality of feature valuesextracted at the extracting step; an aggregating step of aggregating theplurality of feature values extracted at the extracting step into aplurality of classifications based on the distributed representationobtained at the extracting step; an accepting step of accepting aspecification of a classification including a feature value used fordetermining an evaluation criterion for document data among theplurality of classifications aggregated at the aggregating step; and adetermining step of determining the evaluation criterion based on afeature value included in a classification indicated by thespecification accepted at the accepting step.